{
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "orig_nbformat": 2,
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.7.4 64-bit ('base': conda)"
  },
  "interpreter": {
   "hash": "2a7e95a32014fc1ccf24626d45a98c6e7b4373277259c22f47a91d487fc3e8a5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2,
 "cells": [
  {
   "source": [
    "## 数据处理过程\n",
    "数据收集 >> 数据探索/预处理 >> 数据分析/挖掘 >> 结果评价与呈现\n",
    "> 数据预处理:清洗,变换,规约等(需要占用一半时间)  \n",
    ">> 缺失值处理: 删除/填充(固定值,均值,上下数据...)"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "## 数据维度\n",
    "* 一维数据\n",
    ">**列表和数组**  \n",
    ">1. 列表的数据类型可以不同,数组内数据类型一致\n",
    ">> \\[列表\\] = \\[1,{2,3},'字符串'\\]  \n",
    ">> \\[数组\\] = \\[1,2,3,4\\]  \n",
    "\n",
    ">>2.列表和数组均有序  \n",
    ">>>**集合**  \n",
    ">>> {1,2,3,4}  (集合是无序的)\n",
    "\n",
    "* 二维数据\n",
    "**多维列表** \n",
    "\n",
    "* 多维数据\n",
    "**字典的键值对** \n",
    "## 通常用json,XML,Ymal表示数据关系"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "## Numpy 科学计算和数据处理的最基础库\n",
    "## [官方文档](https://numpy.org/doc/stable/reference/arrays.html)  \n",
    "## Numpy 最关键的数据类型:**ndarray**(N维数组对象)  \n",
    "1. 轴(axis,行)和秩(rank/column,列)是ndarray的关键参数\n",
    "2. 支持bool,intc,int32,float64,complex128等元素类型\n",
    "> 元素类型精细定义有助于合理使用存储空间及程序规模  \n",
    "> 所有元素属于同一种类型  \n",
    "> 有丰富的函数  \n",
    "> 相比python列表内存占用更小,运算更快\n",
    "\n",
    "ndarray数组的创建\n",
    "1. python列表，元组赋值\n",
    "2. 使用自带函数创建\n",
    "ndarray数组的属性  \n",
    "ndarray数组的维度变换  \n",
    "ndarray数组的数据类型变换  \n",
    "ndarray数组的操作  \n",
    "ndarray数组的运算  \n",
    "\n",
    "备注:SciPy是开源的基于python的软件生态系统,主要为数学,工程科学服务  \n",
    "[官网](https://scipy.org/)\n",
    "包括 NumPy,SciPy library,Matplotlib,IPython,Sympy,pandas"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ndarray数组的属性\n",
    "import numpy as np\n",
    "\n",
    "lis = [[0,1,2,3,4],[9,8,7,6,5]]\n",
    "a = np.array(lis,dtype=np.int32)  # array:将输入数据转为ndarray类型\n",
    "a?\n",
    "print(a[0,1],'\\n',      # 列表索引\n",
    "a.ndim,'\\n',            # 轴的数量或维度的数量\n",
    "a.shape,'\\n',           # 矩阵对象的尺度 \n",
    "a.size,'\\n',            # 元素的个数\n",
    "a.dtype,'\\n',           # 元素的类型,float64 int32 bool string_等\n",
    "a.itemsize,'\\n',        # 每个元素的大小，字节为单位.int32为4个字节/int8为1个字节\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ndarray数组的创建\n",
    "import numpy as np  # 一种约定\n",
    "\n",
    "# 1. python列表创建\n",
    "lis = [[0,1,2,3,4],[9,8,7,6,5]]\n",
    "# lis = [(0,1,2,3,4),(9,8,7,6,5)]   # 也可\n",
    "a = np.array(lis,dtype=np.int32)    # 为数据类型赋值\n",
    "\n",
    "# 2. 自带函数\n",
    "n = 3\n",
    "shape = (3,2)\n",
    "print(np.arange(n), '\\n',    # 类似range(),返回'一维'0到n-1，默认int32\n",
    "np.ones(shape),'\\n',      # 根据shape生成全1数组，默认float\n",
    "np.zeros(shape,dtype=np.int32),'\\n',    # 根据shape生成全0数组，默认float\n",
    "np.full(shape,2),'\\n',       # 根据shape生成全2数组\n",
    "np.eye(n))              # 生成n*n单位矩阵，对角线为1，其他为0，默认float\n",
    "\n",
    "print(np.ones_like(a),'\\n',\n",
    "np.zeros_like(a),'\\n',\n",
    "np.full_like(a,2))      # 根据a数组的形状生成其他数组\n",
    "\n",
    "b = np.linspace(1,10,4,dtype=np.int32)   # 生成等间距的数组\n",
    "c = np.linspace(1,10,4,endpoint=False)\n",
    "print(b,c)\n",
    "print(np.concatenate((b,c)) )   # 合成数组\n",
    "np.eye(5)   # 创建单位数组"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ndarray数组的维度变换\n",
    "import numpy as np  \n",
    "# 注意：是否改变原数组\n",
    "a = np.ones((2,3,4),dtype=np.int32)\n",
    "b = a.reshape((3,8))    # 保证'元素个数'与原数组一致，不改变原数组\n",
    "c = a.reshape(2,-1)     # 当列数不确定时,用-1替代即可,会自动计算\n",
    "print(\n",
    "    a,'\\n',\n",
    "    b,'\\n',\n",
    "    c,'\\n\\n'\n",
    ")\n",
    "\n",
    "a.resize((3,8))     # 与reshape不同,resize将改变原数组\n",
    "print(a)\n",
    "a.flatten()         # 降维一维数组\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# numpy在运算和内存上的优势\n",
    "import numpy as np\n",
    "myArr = np.arange(1000000)\n",
    "myList = list(range(1000000))\n",
    "%time for _ in range(10): myArr2 = myArr *2     # 对序列内每个数×2\n",
    "%time for _ in range(10): myList2 = [x*2 for x in myList]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### ndarray数组的数据类型变换 .astype\n",
    "import numpy as np  \n",
    "# 注意：是否改变原数组\n",
    "a = np.ones((2,3,4))\n",
    "b = a.astype(np.float)      # 将创建新的数组\n",
    "c = b.tolist()              # 将ndarray转为list数据\n",
    "print(b,'\\n',c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### ndarray数组运算【通用函数 将数组视为一个数对待，不用记，用到的时候查表】\n",
    "# 可用到pandas对象中\n",
    "import numpy as np  \n",
    "## 数组与标量的运算\n",
    "a = np.arange(24).reshape((2,3,4))\n",
    "b = a / a.mean()\n",
    "# 一元函数运算\n",
    "print(np.square(a),'\\n',\n",
    "25-a,'\\n')    # 取平方，未赋值，因此创建新的数组\n",
    "a = np.sqrt(a)  # 平方根，将计算值再赋值给a\n",
    "\n",
    "# 二元函数  +-*/**\n",
    "print(\n",
    "    np.maximum(a,b),'\\n', \n",
    "    np.add(a,b)\n",
    "    )    # 取各元素中较大的一个\n",
    "\n",
    "a < b   # 生成boll数\n",
    "\n",
    "# 不同尺寸之间数组运算 采用广播形式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# *矩阵运算 \n",
    "import numpy as np  \n",
    "## 数组与标量的运算\n",
    "a = np.arange(6).reshape(2,3)\n",
    "b = np.arange(6).reshape(3,2)\n",
    "\n",
    "print(\n",
    "    a.dot(b),'\\n',      # 矩阵点乘积dot,等价于np.dot(a,b)\n",
    "    a.T.dot(a)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### ndarray数组的操作,并非复制,而是在原数组上的操作\n",
    "# NumPy被设计成适合处理非常大的数组，可以想象如果NumPy持续复制数据会引起多少内存问题\n",
    "import numpy as np  \n",
    "# 索引和切片\n",
    "a = np.arange(10)\n",
    "a[1:4] = 1\n",
    "print(\n",
    "    a,'\\n',\n",
    "    a[2],'\\n',  \n",
    "    a[1:10:3],'\\n\\n')   # 从1到9按3的步长递增\n",
    "\n",
    "b = np.arange(12).reshape((3,4))\n",
    "print(b,'\\n',\n",
    "b[0],'\\n',      # 取第一行\n",
    "b[:,[0,2]],'\\n\\n')     # 取第一,三列   \n",
    "\n",
    "c = np.arange(24).reshape((2,3,4))\n",
    "print(b,'\\n',\n",
    "c[1,2,3],'\\n',\n",
    "c[-1,-2,-3])    # 从外到内索引\n",
    "\n",
    "print(c[:,1:3,::2]) # 冒号切片"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 数组表达式替代显式循环的方法，称为向量化\n",
    "\n",
    "# 创建边界值为1,内部值为0的10×10数组\n",
    "import numpy as np  \n",
    "X = np.full((5,5),1)\n",
    "X[1:-1,1:-1]=0    \n",
    "\n",
    "\n",
    "# X[X % 2 == 0] = 3   # np数组内嵌套同样维度的boll数组,对True的元素进行操作\n",
    "# 上行代码将改变原数组X,可以采用 x if condition else y 的方法,即np.where\n",
    "np.where( X%2 == 0,3,6 )   # 将所有偶数改为3,其余改为6 \n",
    "np.where( X%2 == 0,3,X )   # 将所有偶数改为3,其余不变 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[[[0.98197282 0.71980031]\n  [0.86093914 0.56547027]]\n\n [[0.29967169 0.82719313]\n  [0.14028971 0.22420942]]] \n [[[ 0.60532299 -0.39744704]\n  [-1.39343284  0.29753172]]\n\n [[-0.61475509 -0.59855749]\n  [ 0.91394476  0.31615226]]] \n [[1 5 8]\n [8 7 8]]\n"
     ]
    }
   ],
   "source": [
    "### ndarray随机库函数\n",
    "# 伪随机数，因为它们是由具有确定性行为的算法根据随机数生成器中的随机数种子生成的\n",
    "import numpy as np  \n",
    "\n",
    "a = np.random.rand(2,2,2)   # 0-1之间均匀分布的浮点数\n",
    "b = np.random.randn(2,2,2)  # 标准正态分布的浮点数\n",
    "c = np.random.randint(1,10,[2,3])   # 1-10之间[2,3]维度的随机数\n",
    "print(a,'\\n',b,'\\n',c)\n",
    "# 还有一些高级随机数函数，例如choice...\n",
    "# 跟分布有关的函数\n",
    "u = np.random.uniform(0,10,(3,2))   # 从0-10的均匀分布\n",
    "n = np.random.normal(10,5,(3,2))    # 均值为10，方差为5的正态分布"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### ndarray统计函数 , 默认axis=None,所有元素进行计算；给定轴之后，对轴上元素进行计算\n",
    "import numpy as np  \n",
    "a = np.arange(24).reshape((2,3,4))\n",
    "print(a,'\\n')\n",
    "print(np.sum(a),' axis=None\\n',\n",
    "np.sum(a,axis=0),' axis=0\\n',\n",
    "np.sum(a,axis=1),' axis=1\\n',\n",
    "np.sum(a,axis=2),' axis=2\\n',  # 轴/维度的运算较难理解，shape(2,3,4)中维度(axis)从外到内为0,1,2\n",
    "'\\n',\n",
    "np.random.choice(a[0][1],4,replace=True),'\\n',  # 有放回的采用:replace=True\n",
    "np.mean(a,axis=1,keepdims=True),'期望\\n',   # 期望，对第一维度进行运算,保持维度方便计算\n",
    "np.average(a,axis=0,weights=[1,2]),'加权平均\\n',   # 加权平均\n",
    "np.std(a),'方差\\n',\n",
    "np.var(a),'标准差\\n',\n",
    "'\\n',\n",
    "np.max(a),'最大值\\n',\n",
    "np.ptp(a),'最大值-最小值\\n',\n",
    "np.median(a),'中位数\\n'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 数组排序\n",
    "import numpy as np  \n",
    "a = np.arange(24,0,-1).reshape((2,3,4))\n",
    "print(\n",
    "    a,'\\n\\n',  # 原数组\n",
    ")\n",
    "a.sort(0) \n",
    "print(\n",
    "    a,'\\n\\n',  # 将第一个维度所有元素排序\n",
    ")\n",
    "\n",
    "a = np.arange(24,0,-1).reshape((2,3,4))\n",
    "a.sort(1) \n",
    "print(\n",
    "    a,'\\n\\n',  # 将第二个维度所有元素排序,可理解为:将第一维度下的数组所有列进行排序\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[ 0.38266818 -0.51878319 -1.52094612  0.26232321  1.2811646   0.45997137\n -0.31969225  0.70266453  1.58355447  0.77030692] 7 True False\n"
     ]
    }
   ],
   "source": [
    "# 布尔数组的方法\n",
    "import numpy as np\n",
    "arr = np.random.randn(10)\n",
    "bools = arr > 0\n",
    "print(\n",
    "arr,\n",
    "bools.sum() ,    # True的个数\n",
    "bools.any() ,    # 是否至少一个True\n",
    "bools.all()     # 是否全为True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### ndarray梯度函数：连续值之间的变化率，即斜率\n",
    "import numpy as np\n",
    "a = np.random.randint(0,20,(5))\n",
    "print(a,'\\n',\n",
    "np.gradient(a),'\\n',    # 梯度值：(后一个数-前一个数)/距离\n",
    ")\n",
    "b = np.random.randint(0,20,(3,4))\n",
    "print(b,'\\n',\n",
    "np.gradient(b),'\\n',    # n维数组返回n个维度的梯度值\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### ndarray数组的存取\n",
    "# 注意：csv只能存取一维、二维数据\n",
    "import numpy as np  \n",
    "\n",
    "a = np.arange(100).reshape(5,20)\n",
    "# 写入csv文件 \n",
    "np.savetxt('npa.csv',a,fmt='%d',delimiter=',')  # fmt为主要需要修改的参数\n",
    "# 读入csv文件\n",
    "b = np.loadtxt('npa.csv',dtype=np.float,delimiter=',',unpack=False)\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 多维数据的存取\n",
    "import numpy as np  \n",
    "\n",
    "a = np.arange(100).reshape(5,10,2)\n",
    "# 写入dat文件\n",
    "a.tofile('npb.dat',sep=',',format='%d') # 失去维度信息，不设置sep将存入二进制数据\n",
    "# 读取dat文件\n",
    "c = np.fromfile('npb.dat',dtype=float,count=-1,sep=',') # count为读取个数，=-1表示读入整个文件的数据\n",
    "print(c.reshape(5,10,2))    # 还原维度信息\n",
    "\n",
    "np.save('npa.npy',a)    # 以npy数据格式，存入元素和维度信息\n",
    "b = np.load('npa.npy')\n",
    "print(b)    # 维度信息被还原"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ndarray的函数是在C语言级别实现的,运算速度极快\n",
    "# 在数据量较大时比较math函数和ndarray函数的运算速度\n",
    "import time\n",
    "import math\n",
    "import numpy as np\n",
    "\n",
    "x = np.arange(0,100,0.01)\n",
    "t_m1 = time.process_time()\n",
    "for i,t in enumerate(x):\n",
    "    x[i]=math.pow((math.sin(t)),2)\n",
    "t_m2 = time.process_time()\n",
    "print('Running time of math:{:.2f}'.format(t_m2-t_m1))\n",
    "\n",
    "y = np.arange(0,100,0.01)\n",
    "t_n1 = time.process_time()\n",
    "y = np.power(np.sin(y),2)\n",
    "t_n2 = time.process_time()\n",
    "print('Running time of numpy:{:.2f}'.format(t_n2-t_n1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### 图像的数组表示\n",
    "# 采用RGB色彩模式，每个像素点由RGB色彩通道构成,其中R[0-255]G[0-255]B[0-255]\n",
    "# 通过PIL库处理图像，Image类代表一个图像\n",
    "# 图象是由像素组成的二维矩阵(X,Y)，每个元素(像素点)是一个RGB值(R,G,B)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 图像变换\n",
    "from PIL import Image\n",
    "import numpy as np\n",
    "\n",
    "jpg = Image.open(r'C:/Users/ZSY/Pictures/Camera Roll/2.jpg')\n",
    "im = np.array(jpg)\n",
    "print(im.shape,im.dtype)    # 图像=三维数组：高度，宽度，像素RGB值\n",
    "\n",
    "b = 255 - a\n",
    "im2 = Image.fromarray(b.astype('uint8'))\n",
    "im2.save(r'C:/Users/ZSY/Pictures/Camera Roll/2_反向.jpg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 手绘效果图\n",
    "from PIL import Image\n",
    "import numpy as np\n",
    "\n",
    "a = np.asarray(Image.open(r'C:/Users/ZSY/Pictures/Camera Roll/2.jpg').convert('L')).astype('float')\n",
    "\n",
    "depth = 10. \t\t\t\t\t\t# (0-100)\n",
    "grad = np.gradient(a)\t\t\t\t#取图像灰度的梯度值\n",
    "grad_x, grad_y = grad \t\t\t\t#分别取横纵图像梯度值\n",
    "grad_x = grad_x*depth/100.\n",
    "grad_y = grad_y*depth/100.\n",
    "A = np.sqrt(grad_x**2 + grad_y**2 + 1.)\n",
    "uni_x = grad_x/A\n",
    "uni_y = grad_y/A\n",
    "uni_z = 1./A\n",
    "\n",
    "vec_el = np.pi/2.2 \t\t\t\t\t# 光源的俯视角度，弧度值\n",
    "vec_az = np.pi/4. \t\t\t\t\t# 光源的方位角度，弧度值\n",
    "dx = np.cos(vec_el)*np.cos(vec_az) \t#光源对x 轴的影响\n",
    "dy = np.cos(vec_el)*np.sin(vec_az) \t#光源对y 轴的影响\n",
    "dz = np.sin(vec_el) \t\t\t\t#光源对z 轴的影响\n",
    "\n",
    "b = 255*(dx*uni_x + dy*uni_y + dz*uni_z) \t#光源归一化\n",
    "b = b.clip(0,255)   # 避免数据溢出\n",
    "\n",
    "im = Image.fromarray(b.astype('uint8')) \t#重构图像\n",
    "im.save(r'C:/Users/ZSY/Pictures/Camera Roll/2_手绘.jpg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ]
}